Nml-optimal Histogram Density Estimation

Density estimation is one of the central problems in statistical inference and machine learning. Given a sample of observations, the goal of histogram density estimation is to find a piecewise constant density that describes the data best according to some pre-determined criterion. Although histograms are conceptually simple densities, they are very flexible and can model complex properties like multi-modality with a relatively small number of parameters. Furthermore, one does not need to assume any specific form for the underlying density function: given enough bins, a histogram estimator adapts to any kind of density. Most existing methods for learning histogram densities assume that the bin widths are equal and concentrate only on finding the optimal bin count. These regular histograms are, however, often problematic. It has been argued [2] that regular histograms are only good for describing roughly uniform data. If the data distribution is strongly non-uniform, the bin count must necessarily be high if one wants to capture the details of the high density portion of the data. This in turn means that an unnecessary large amount of bins is wasted in the low density region. To avoid the problems of regular histograms one must allow the bins to be of variable width. For these irregular histograms, it is necessary to find the optimal set of cut points in addition to the number of bins, which naturally makes the learning problem essentially more difficult. For solving this problem, we regard the histogram density estimation as a model selection task, where the cut point sets are considered as models. In this framework, one must first choose a set of candidate cut points, from which the optimal model is searched for. The quality of each of the cut point sets is then measured by some model selection criterion. Our approach is based on information theory, more specifically on the Minimum description length (MDL) principle developed in the series of papers [3, 4, 5]. MDL is a well-founded, general framework for performing model selection and other types of statistical inference. The fundamental idea behind the MDL principle is that any regularity in data can be used to compress the data, i.e., to find a description or code of it such that this description uses the least number of symbols, less than other codes and less than it takes to describe the data literally. The more regularities there are, the more the data can be compressed. According to the MDL principle, learning can be equated with finding regularities in data. Consequently, we can say that the more we are able to compress the data, the more we have learned about it. Model selection with MDL is done by minimizing a quantity called the stochastic complexity, which is the shortest description length of a given data relative to a given model class. The definition of the stochastic complexity is based on the normalized maximum likelihood (NML) distribution introduced in [6, 5]. The NML distribution has several theoretical optimality properties, which make it a very attractive candidate for performing model selection. It was originally [5] formulated as a unique solution to the minimax problem presented in [6], which implied that NML is the minimax optimal universal model. Later [7], it was shown that NML is also the solution to a related problem involving expected regret. See [8, 9] for more discussion on the theoretical properties of the NML.